FAQ: Why do the measurements returned from my unit seem more accurate than the specified measurement resolution?

The simple answer is that measurement accuracy and measurement resolution are not the same thing. Each time a measurement is taken the result is always rounded to the closest resolution value. So why do some measurements seem to have a better resolution that what is specified when they are measuring a constant current?

First of all, the measurement accuracy specification is always worse than the measurement resolution. Because the accuracy is always worse than the resolution, each measurement value (or point) can be anywhere within the + and - accuracy band. When these measurement points are averaged and returned, the returned values may just happen to be very close together - and appear better than the resolution of the measurement. This would be a coincidence, it does not mean that the measurement is better than the resolution.

Refer to the blog powersupply.blogs.keysight.com/2012/08/power-supply-resolution-versus-accuracy.html for more information.

Applies to: N6700A, N6700B, N6701A, N6702A, N6705A, N6705B, N6710B, N6711A, N6712A, N6715A, N6715B, N6731B, N6732B, N6733B, N6734B, N6735B, N6736B, N6741B, N6742B, N6743B, N6744B, N6745B, N6746B, N6773A, N6774A, N6775A, N6776A, N6777A, N6751A, N6752A, N6753A, N6754A, N6755A, N6756A, N6761A, N6762A, N6763A, N6764A, N6765A, N6766A, N6781A, N6782A, N6783A, N6784A, N6785A, N6786A

Applies to: N6950A, N6951A, N6952A, N6953A, N6954A, N6970A, N6971A, N6972A, N6973A, N6974A, N6976A, N6977A

Applies to: N7950A, N7951A, N7952A, N7953A, N7954A, N7970A, N7971A, N7972A, N7973A, N7974A, N7976A, N7977A

Applies to: N3300A, N3301A, N3302A, N3303A, N3304A, N3305A, N3306A, N3307A

Select Home to return to the table of contents for more FAQs.

Visit our Power Products Forum for additional topics and discussions.

Read technical postings on our Power Blog – Watt’s Up?